Multiagent Coordination in Cooperative Q-learning Systems

Many reinforcement learning architectures fail to learn optimal group behaviors in the multiagent domain. Although these coordination difficulties are often attributed to the non-Markovian environment created by the gradually-changing policies of concurrently learning agents, a careful analysis of the situation reveals an underlying problem structure which can cause suboptimal group policies even when the Markovian properties of the learning environment are preserved. This underlying structure is termed the multiagent coordination problem, and it can be viewed as a combination of two related but distinct limitations of cooperative multiagent learning systems: action shadowing and joint action prediction. This paper discusses the causes of each of these limitations and their effects on systems of cooperative Q-learning agents, including the conditions which must be met in order to guarantee the execution of optimal group policies. Multiagent coordination strategies presented by other researchers are considered from the perspective of this new problem framework, and an algorithm is presented which extends some of these coordination strategies to improve the tractability of large-scale multiagent learning.